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A. Preface 



Children who come from cultural and linguistic minority backgrounds often founder in 
American schools. Many do not gain a solid grounding in English reading and writing or in 
mathematics and science by the time they enter high school. As young adults they are 
inadequately prepared for higher education or for all but the most menial employment. 

This situation is unacceptable. The challenge of educating language minority students to the 
high standards we expect of all children is not well understood. It is nonetheless clear that little 
progress will be made unless the educational and organizational practices at many of today's 
public schools are reformed. 

In 1990, the Office of Educational Research and Improvement (OERI) of the US Department 
of Education issued an RFP to identify and study exemplary school reform efforts involving the 
education of language minority students. The RFP directed the study to focus on language arts in 
grades 4 through 6 and math/science in grades 6 through 8. It is neither desirable nor possible to 
separate the LEP program at these grade levels from the entire experience of students at a school. 
Therefore, the study analyzes the context of school reform and how that school reform affects the 
entire curriculum and program of instruction for LEP students. 

This document, Volume III: Technical Appendix, presents the research design and methodology 
of the Student Diversity Study. It is part of a set of three reports on the project. Volume I: Findings 
and Conclusions presents the major findings of ihe study, a literature review, summary of 
methodology, case study summaries, a cross site analysis and policy and research implications 
emerging from the study. Detailed case studies on eight exemplary schools are presented in Volume IT. 
The Case Studies. The study also commissioned research papers, which have been edited and 
published as a book. Language and Learning: Educating Linguistically Diverse Students 
(Beverly McLeod, editor, SUNY Press, 1994). 

OERI has funded eleven other companion studies that examine different aspects of school 
reform The entire set of reports will thus provide a comprehensive description and analysis of 
.eform from the empirical perspective of outstanding practices in the field. 
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E. Research Design 



The goal of the Student Diversity Study was to identify and describe exemplary school reform 
efforts for language minority students in grades 4 through 8 in three curricular areas: language arts, 
science, and mathematics. The focus on language arts curriculum is directed at grades 4 through 6 and 
the focus on mathematics and science curriculum is in grades 6 through 8. The study team identified 
and described eight exemplary schools which offer state-of-the-art curriculum and instruction in one or 
more of the three curricular areas in a restructured school. The goal was to synthesize elements of 
models of exemplary programs. More specifically, this research identified theory-based and 
practice-proven models to effectively teach language arts, math, and science to students fi-om linguistic 
and cultural diverse backgrounds. These models enable educators to help such students overcome the 
linguistic, cognitive, and social psychological barriers that prevent them fi-om achieving to their full 

potential. 

While limitations of study resources made it impossible to include sites which represent a wide 
range of contexts (such as demography, geographic region, and language mix), schools were selected 
to reflect a variety of contexts. Schools selected for the study demonstrated innovative, high quality 
cutriculum in a reformed school context, as well as excellent language development programs for LEP 
students. In addition, case study sites implemented innovative school reform approaches beyond the 
standard observed in excellent but otherwise traditional school settings. 

Areas of Inquiry 

The RFP identified three areas for specific inquiry for this study: 

1 Design of effective instructional strategies for culturally and linguistically diverse students; 

2. Implementation of those strategies under various conditions; and 

3, Impact of those strategies on rtudents. 

The research team developed a series of Analysis Questions that address each of these three areas 
of inquiry. The three areas of inquiry and the corresponding Analysis Questions are presented 
below 

Tables E- 1 through E-3 match the Analysis Questions with the data sources or analytical 
approaches that address each question. The data sources or analytical approaches, which are 
described more fully in the remainder of tliis document, are literature review, commissioned 
papers, site selection, case studies, and cross-site analysis. Please notice that the analysis 
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questions listed in the table will be addressed using a variety of different data sources or analytical 
approaches. A later section focuses on the central data source, the case studies, and presents case 
study questions for addressing these areas of inquiry. 

1. Design. Research into the design of effective instructional strategies required the research 
team to identify, examine, Ciialyze, evaluate, and describe these practices as they are exhibited in 
exemplary schools and c'assrooms. Table E-1 lists a series of revised Analysis Questions related 
to program design and matches them with the data sources or analytical approaches that address 
the analysis questions. 

2. Implementation. Many reforms and models fail live up to their promise when implemented 
under the real conditions of American schools. Consequently, the study will identify exemplary 
practices that have been implemented and demonstrated their effectiveness over a period of time. 
Our field investigations will focus on uncovering factors and conditions that helped bring the 
program into being or had to be overcome to achieve success. Table E-2 presents Analysis 
Questions about implementation of reform along with the data sources or analytical approaches 
that informed the questions. 

3. Impact. A major challenge for this research was to collect data about how new instructional 
approaches affect student learning and other measures of the program's impact. The duration of 
this research project was too short, and the budget was too limited, to conduct a longitudinal 
study of student outcomes. Moreover, we were pe.ssimistic about the possibility of gathering data 
to allow comparison of student outcomes across sites for several reasons. LEP student test 
scores often are hard to come by in schools and are generally not comparable across sites because 
LEP students are often not given the standardized tests that districts or states require of most 
students. The transiency r d mobility of LEP students is another factor that makes comparable 
data very difficult to obtain. Therefore, the study could not demonstrate quantitatively that the 
eight case study sites are exemplary in the sense of demonstrated evidence of significantly higher 
student achievement scores. Nevertheless, the nomination, screening, and field visits ail led to the 
conclusion that these schools were highly infiovative and followed practices that are considered by 
researchers to provide outstanding learning opportunities for LEP — and all — students. Table E-3 
lists Analysis Questions about both student and program outcomes. 
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Table E-1 

Analysis Questions Related to Design 
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Analysis Questions Related to Implementation 
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Table E-3 

Analysis Questions Related To Impact 



Data Sources or Analytical Approaches 
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Case 

Studies 
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Site 

Selection 
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Commissioned 

Papers/ 

National 

Conference 














Literature 

Review 


X 


X 








X 




Analysis Questions 


1 . How do reform programs assess llieir own progress in improving 
student IcarninE? 


2. Do schools have evidence lhal student learning outcomes improved? 


3. What is the overall assessment of the program, its strengths and 
weaknesses? 


4. How can assessments be used to reHne reforms? 


S. What are the anticipated and unanticipated benefits derived from 
and difficulties encountered in the implementation of reform 
nroerams? 


6. What was the role of research, research-based knowledge, and other 
information in program assessment? 
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Design of Research Activities 

This section discusses the design of the study’s four principle research activities: 1) literature 
review, 2) commissioned papers and national conference, 3) site selection, and 4) case studies. 

The methodology employed for the site selection, cases studies, and cross-site analysis is 
described in the subsequent section entitled Methodology. 

1. Literature Review. The research team conducted an extensive review of the literature for the 
study. The review has four main foci: 1) Strategies for teaching culturally and linguistically 
diverse populations, including cooperative and group learning activities and instructional 
conversation; 2) Second language acquisition and language arts instructional strategies especially 
for upper elementary grades; 3) Science and math instructional strategies, particularly at the 
middle school level; 4) Theory and practice of school reform. Project staff have examined the 
mainstream literature in these areas, as well as reports produced but not published by ongoing 
research projects and publications by individual school districts. 

The literature review served as a crucial underpinning of our examination of the research 
questions posed by the study. Additionally, the literature review served as input to the criteria for 
the selection of sites for the case studies. Finally, the literature review provided important 
information that informed the team's assessment of the impact of reform activities in the area of 
education of culturally and linguistically diverse students. 

2. Commissioned Papers and National Conference. The research papers commissioned for 
this study advanced the study's work by exploring critical issues in the relationship between school 
reform and improvement of educational outcomes for linguistically and culturally diverse students. 
The papers were reviewed at the national conference by researchers, policymakers, and 
practitioners. Through the papers and conference proceedings, the study team gained insight 
from national experts that informed the site selection, data collection, and data analysis processes. 
The research papers were edited and published by the State University of New York Press as 
Language and Learning: Educating Linguistically Diverse Students (1994). The paper titles and 
authors follow. 

Education Reform 

• Linguistic Diversity and Academic Achievement, Beverly McLeod 

• The Impact of the Education Teform Movement on Limited English Proficient 
Students, Patricia Gandara 

• The Role of Discourse in Learning, Schooling, and Reform, Hugh Mehan 




Culture and Learning 

• The Value of a Multicultural Education for All Students, Christine E. Sleeter 

• Research Knowledge and Policy Issues in Cultural Diversity and Education, Roland Tharp 
Language and Literacy 

• First and Second Language Literacy in the Late Elementary Grades, Barry McLaughlin 

• Teaching Strategies: Their Possibilities and Limitations, Lilia I. Bartoicnie 
Math and Science 

• A Communication Framework for Mathematics: Exemplary Instruction for Culturally and 
Linguistically Diverse Students, Mary E. Brenper 

• Language Diversity and Science Learning: The Need for a Critical System of Meaning, 
Alejandro J. Gallard and Deborah J. Tippins 

3. Site Selection. The study team viewed the site nomination process both as a method of 
identifying sites for case studies but also as an additional source of data on exemplary programs 
for LEP students. Figure 1 represents in graphic form the study team's design for site selection 
which involves five stages; identification of a pool of nominees, initial paper screening, in-depth 
telephone screening to narrow pool of potential fieldwork sites, selection of preliminary fieldwork 
sites, and selection of case study sites. Each of these stages are iescribed in detail in Section F, 
Methodology under the header Sample Selection. 

Figure £-1 
The Sample Design 
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Pool of 156 Nominated Sites 



Telephone Screening of 75 Candidate Sites 



25 Potential Fieldwork Sites 

1 

15 Fieldv'ork Sites 



8 Case Study Sites 



4. Case Studies. The heart of the research activities was the case studies. The following material 
describes our approach to the case studies, details the research questions that provide a 
framework for the field instruments and cross-walks these research elements to the analysis 
questions presented in the preceding section. 

The case study research design included state, district, and school interviews, focus groups 
discussions, classroom observations, sclv '1 observations, and a document collection. Fieldwork 
staff used these sources to gather data on a series of research questions, The fieldwork research 
questions are contained on the following pages. They have been organized around five Case 
Study Research Questions; 

1 . What is the context for reform? 

2. What is the design of the reform and how is it implemented? 

3 What is th. role of research-based information on the reform? 

4 What, resout ces are required for the reform ? 

5 What is the impact of the reform? 

These five Case Study Research Questions are derived from the three Areas of Inquiry and 
Analysis Questions discussed above. It proved to be convenient to reorganize the questions for 
the purpose of the case study work and the design of the case study instruments. Table E-4 
displays the crosswalk between the Case Study Research Questions and the Analysis Questions 
(shown in Tables E-1, E-2, and E-3). 

The case study instruments were designed using a three step procedure. First, we 
operationalized each Case Study Research Question into a range of data elements that, taken 
together, would answer the Research Question. Second, we cross checked these operational data 
elements with the Analysis Questions to make sure that the data collected would address the 
Analy''S Questions. Third, for each operational data element, we developed specific items for the 
Data Collection Instruments. Tables E-5 through E-9 display the results of these steps. These 
tables show' that each Case Study Research Question has been elaborated with Sub-Research 
Questions that appear in the left column of the tabl«';; the middle column contains Operational 
Elements for Data Collection aboi * which the team collected data during the fieldwork. The 
column on the right shows the corresponding data source that was used to gather the information 
regarding each Operational Element 
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Table E-4 

Crosswalk of Case Study Research Questions and 
Analysis Questions Related to Design, Implementation, and Impact 



Case Study Research Questions 


Corresponding Analysis Questions 


1. What is the context for reform? What factors 
helped to initiate, develop, and sustain reform? 
What were the major barriers to reform and 
how were they overcome? What are unique 
oroerammatic and demographic conditions? 


1. Design: 8-10 

2. Implementation: 1-6, 10, 11, 14 


2. What is the design of the reform and how is 
it implemented? 

Restructuring: What elements of restructuring 
are in place? How is the program organized and 
governed? Hew is the program staffed? How 
are children grouped for instruction? 
Curriculum: What type of curriculum is used? 
How is it integrated across content areas? How 
is it developed? 

Language Acquisition Strategies: What 
strategies are used for language acquisition? 
How does the program build on the cultures of 
the students? 


1. Design: 1-12 

2. Implementation: 4, 7-9 


3. What is the role of research-based 

information on the reform? What has been 
the rc ? of research and research-based 
information in designing, implementing, and 
evaluating the reform? the language acquisition 
strategies? the curriculum? 


2. Implementation: 13 

3. Impact: 6 


4. What resources are required for the reform? 
What were the sources of financial support for 
the program: federal, state, local, or private? 
What is the approximate cost of the reform; 
development, operation, evaluation, and 
training? 


2. Implementation: 12 


5. What is the impact of the reform? What has 
been the approach to assessing the student 
learning outcomes from the reform? What are 
the results of the assessment? W^at are the 
strengths and weaknesses of the reform? 


3. Impact: 1-6 



Table E-5 

Crosswalk between Case Study Research Question #1 and Data Sources 
Case Study Research Question #1: What Is The Context for Reform? 



Sub-Research 

Questions 


Operational Elements for 
Data Collection 


Corresponding Data Sources 




A. Impetus for and history of 
restructuring 


State Director of Second Languages Interview 
Assistant Superintendent Interview 
District Second Language Director Interview 
District Curriculum Director Interview 
Principal Interview 


B. Leadership in reform 
movement 


State Director of Second Languages Interview 
Assistant Superintendent Interview 
District Second Language Director Interview 
District Curriculum Director Interview 
Principal Interview 
Teacher Interview 
External Partner Interview 


What factors helped 
to initiate, develop, 
and sustain reform? 
\Mtat were the major 
barriers to reform 
and how were they 
overcome? What are 
unique 

programmatic and 

demographic 

conditions? 


C. Incentives for reform 


State Director of Second Languages Interview 
Assistant Superintendent Interview 
Principal Interview 


D. Policies and funding 
priorities related to reform 


State Director of Second Languages Interview 
Disi /ict Curriculum Director Interview 
Principal Interview 


E. Efforts toward improving 
curriculum and instruction in 
language arts, math, and/or 
science 


State Director of Second Languages Interview 
Assistant Superintendent Interview 
District Second Language Director Interview 
District Curriculum Director Interview 
Principal Interview 
Teacher Interview 


F. Demographic and 
economic conditions 
affecting the reform 


Assistant Superintendent Interview 
District Second Language Director Interview 
Principal Interview 
Document Checklist 


G. Desegregation policies 
affecting the reform 


State Director of Second Languages Interview 
District Second Language Director Interview 
Principal Interview 
Document Checklist 


H. Role and attitudes of school 
staff toward the reform 


Principal Interview 
Teacher Interview 


I. Role of external partners in 
the reform 


Assistant Superintendent Interview 
Principal Interview 
Teacher Interview 
External Partner Interview 


J. Federal, state, and district 
role in the reform 


State Director of Second Languages Intervic\^ 
Assistant Superintendent Interview 
District Second Language Director Interview 
District Curriculum 1 ! rector Interview 
Principal Interview 
External Partner Interview 


K. Piuent and community 
involvement in the reform 


Principal Interview 
Teacher Interview 

District Second language Director Interview 
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Table E-6 

Crosswalk between Case Study Research Question U2 and Data Sources 
Case Study Research Question #2: 

What Is the Design of the Reform and How Is It Implemented? 



Sub-Research 

Questions 


Operational Elements for 
Data Collection 


Corresponding Data Sources 




A. Organization of teaching 
(e.g., team teaching) 


District Curriculum Director Interview 
Principal Interview 
Teacher Interview 


Restructuring: 


B. Use of time 


District Second Language Director Interview 
District Curriculum Director Interview 
Principal Interview 
Teacher Interview 


What elements of 
restructuring are in 
place? How is the 
program organized 
and governed? 


C. Governance structure 


District Curriculum Director Interview 
Principal Interview 
Teacher Interview 
Parent Focus Group Protocol 
Document Checklist 


How is the program 
staffed? How are 
children grouped 


D. School climate 


Student Focus Group Protocol 
School Observation 
Document Checklist 


for instruction? 


E. Integrated services 


Principal Interview 
Parent Focus Group Protocol 
School Observation 




F. Leadership 


District Second Language Director Interview 

Principal Interview 

Teacher Interview 

Parent Focus Group Protocol 

External Partner Interview 

School Observation 




G. Modifications to physical 
environment to support 
school organization 


School Observation 




H. Student placement policies 
and practices 


District Second Language Director Interview 
Principal Interview 




I. Student grouping mthin 
classroom 


Teacher Interview 




J. Classroom instructional 
approaches 


Teacher Interview 
Student Focus Group Protocol 




K . Teacher and administrator 
qualifications 


District Second Language Director Interview 
Principal Interview 
Teacher Interview 




L. Staff development policies 
and practices 


District Second Language Director Interview 
Assistant Superintendent Interview 
District Curriculum Director Interview 
Principal Interview 
Teacher Interview 




M. Use of paraprofcssionals 


District Second Language Director Interview 
Principal Interview 
Teacher Interview 



Table E-6 (c 


lont.l 


Sub'Rescarch 

Questions 


Operational Elements for 
Data Collection 


Corresponding Data Sources 


Restructuring (cont.) 


N. Role of external paruier in 
program organization 


Principal Interview 
Teacher Interview 
External Partner Interview 
School Observation 


Curriculum: What 
type of curriculum is 
used? How is it 


0. Language ans, math, and/or 
science curriculum 


Assistant SuperiMtendent Interview 

District Second Language Director Interview 

District Curiculum Director Interview 

Principal Interview 

Teacher Interview 

Student Focus Group Protocol 


integrated across 
content areas? How 
is it developed? 


P. Integrated curriculum 


District Curriculum Director Interview 
Principal Interview 
Teacher Interview 




Q. Use of instructional 
technology 


Principal Interview 
Teacher Interview 
Student Focus Group Protocol 
School Observation 




R. R >le of external partner in 
curriculum development and 
instruction 


District Curriculum Director Interview 
Principal Interview 
Teacher Interview 
External Partner Interview 


Language 

Acquisition 

Strategies: 

What strategies are 
used for language 
acquisition? How 
does the program 
build on the cultures 
of the students? 


S. Language acquisition 
strategies 


District Second Language Director Interview 
Principal Interview 
Teacher Interview 


T. Approach to transition to 
English 


District Second Language Director Interview 
Principal Interview 
Teacher Interview 


U. Language use in classroom 


District Second Language Director Interview 

Principal Interview 

Teacher Interview 

Parent Focus Group Protocol 

Student Focus Group Protocol 


V. Instructional practices for 
recent immigrants 


District Second Language Director Interview 
Principal Intervieur' 

Teacher Interview 


W. LEP student designation 
policies and procedures 


District Second Language Director Interview 
Principal Interview 


X. Parent and community 
involvement 


District Second Language Director Interview 

Principal Interview 

Teacher Interview 

Parent Focus Group Protocol 

School Observation 


Y. School efforts to validate 
students' culture 


Principal Interview 
Teacher Interview 
Parent Focus Group Protocol 
School Observation 
Document Checklist 


Z School and classroom 
physical environment 


School Observation 
Classroom Observation 
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Table E-7 

Crosswalk between Case Study Research Question #3 and Data Sources 
Case Study Research Question #3: 

What Is the Role of Research-based Information in the Reform? 



Sub-Research 

Questions 


Operational Elements for 
Data Collection 


Corresponding Data Sources 


^hat has been the 
role of research 
and research-based 
information in 
designing, 
implementing, and 
evaluating the 
reform? Me 
language 
acquisition 
strategies? the 
curriculum? 


A. Research on instructional 
approach 


Principal Interview 


B. Research on organization of 
teaching (e.g.. team teaching) 


Principal Interview 


C. Research on use of time 


Principal Interview 


D. Research on assessment 


Principal Interview 

District Evaluation Director Interview 


E. Research on language 
acquisition strategies 


Principal Interview 

District Second Language Director Interview 


F. Research related to student 
placement and grouping 


Principal Interview 

District Second Language Director Interview 


G. Research on language arts, 
math, and/or science 
corriculum 


Principal Interview 

District Curriculum Director Interview 


H. Extenud partners' research 
base 


External Partner Interview 
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Table E-8 

Crosswalk between Case Study Research Question #4 and Data Sources 
Case Study Research Question #4: What Resources Are Required for the Reform? 



Sub-Research 

Questions 


Operational Elements for 
Data Collection 


Corresponding Instruments (Question 
Numbcn) 


What were the 
sources of financial 
support for the 
program: federal, 
state, local, or 
private? What is the 
approximate cost of 
the reform: 
development, 
operation, 
evaluation, and 
training? 


A. Funding policies for LEP 
student programs 


State Director of Second Languages Interview 
Assistant Superintendent Interview 
District Second Language Director Interview 
District Business Officer Interview 
Principal Interview 
Document Checklist 


B. Resources for LEP student 
programs 


District Second Language Director Interview 
District Business Officer Interview 
Principal Interview 
Document Checklist 


C. Language arts, math, and/or 
science program policies, 
funding priorities, and 
resources 


State Director of Second Languages Interview 
Assistant Superintendent Interview 
District Curriculum Director Interview 
District Business Officer Interview 
Principal Interview 
Document Checklist 


D. Assessment policies, funding 
priorities, and resources 


District Evaluation Director Interview 
District Business Officer Interview 
Principal Interview 
Document Checklist 


E. StalSing policies, funding 
priorities, and resources 


District Business Officer Interview 
Principal Interview 
Document Checklist 


F. Instructional material 
policies, funding 
priorities and 
resources 


District Curriculum Director Interview 
District Business Officer Interview 
Principal Interview 
Document Checklist 


G. Instructional technolog} 
policies, funding priorities, 
and resources 

H. External funding 


District Curriculum Director Interview 
District Business Officer Interview 
Principal Interview 
Document Checklist 


District Business Officer Interview 
Principal Interview 
Document Checklist 
External Partner Interview 
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Table E-9 

Crosswalk between Case Study Research Question #5 and Data Sources 
Case Study Research Question #5: What Is the Impact of the Reform 



Sub-Research 

Questions 


Operational Eiements for 
Data Coiiection 


Corresponding Data Sources 




A. Student assessment policies 


State Director of Second Languages Interview 
District Curriculum Director Interview 
District Evaluation Director Interview 
Principal Interview 
Document Checklist 




B. Student assessment methods 


District Curriculum Director Interview 


What has been the 
approach to assessing 
the student learning 
outcomes from the 


and strategies 


District Evaluation Director Interview 
Principal Interview 
Teacher Interview 
Document Checklist 


reform? What are the 


C. Student assessment linked to 


District Curriculiun Director Interview 


results of the 
assessment? What 
are the strengths and 
weaknesses of 


Student outcomes 


District Evaluation Director Interview 
Principal Interview 
Teacher Interview 
Document Checklist 


the reform? 


D. Student attendance 


Principal Interview 
Document Checklist 




E. Student academic performance 


Document Checklist 




F. Student suspensions and 
expulsions 


Document Checklist 




G. Program evaluation 


District Evaluation Director Interview 
Principal Interview 
External Partner Interview 
Parent Focus Group 
Student Focus Group 
Document Checklist 




H. LEP Student Redesignations 


District Second Language Director Interview 
Principal Interview 
Document Checklist 




I. Strengths and weaknesses of 
the language acquisition 
strategics 


State Director of Second Languages Interview 
District Second Language Director Interview 
Principal Interview 
Teacher Interview 
Parent Focus Group 




J. Strengths and weaknesses of 
the language arts, math, 
and/or science curriculum 


Assistant Superintendent Interview 
District Curriculum Director Interview 
Principal Interview 
Teacher Interview 
Parent Focus Group 




K. Strengths and weaknesses of 


Principal Interview 


1 


the restructured environment 


Teacher Interview 
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F. Methodology 



This section discusses the methodology used in the Student Diversity Study. The methods used in 
sample selection, data collection in the intensive case study visits, and cross-site analysis are described. 

Sample Selection 

Site Nomination Process. The team began the process by identifying a pool of nominations 
which involved four stages: 1) contacting key informants, 2) developing a comprehensive list of 
nomination sources, 3) contacting those sources, and 4) following up as necessary. Study team 
members spoke by phone and in person with a group of key informants, researchers and 
practitioners from across the country and asked for advice on particular sites, on others to 
contact, and on the state of the art in schools and districts. The study team developed a large list 
of potential sources in the following categories to contact for nominations. 

• Directors of Federally-funded Centers 

• Advisors to the National Center for Research on Cultural Diversity and Second Language 
Learning 

• Curriculum Groups and Associations (National Association of Bilingual Education, Center for 
Applied Linguistics, etc.) . 

• Project Advisors 

• State Second Language Directors (in the 10 states with the largest LEP population’ ) 

• Commissioned Paper Authors 

• School District Second Language Directors (in districts with high concentrations of LEP 
students) 

• National Diffusion Network Coordinators (in 20 states with the highes : concentration of LEP 
students) 

• Individuals Suggested by Key Contacts 

• Presenters at the California Association of Bilingual Education and National Association of 
Bilingual Education conference with relevant topics 

• State Title VII Directors (in all states) 



' These state arc Arizona, California, Florida Illinois, Massachusetts, Ne>\’ Jersej’, New Mr. xico, New York, 
Pennsylvania and Te.xas. 



A letter describing the project, outlining the types of sites we sought was sent to those in 
each of the above mentioned groups. A nomination form was included with each letter. A total 
of 350 letters were mailed. In addition, nomination forms were distributed to participants at the 
project's national conference. State staff were asked to identify key individuals in universities who 
train bilingual teachers, intermediate units who do teacher training, districts with innovative 
programs and other key contacts. 

Staff p’aced follow up calls to state second language directors in the 20 states with the largest 
LEP population, project advisors, paper authors, and to selected members of other groups. In 
some cases, informants nominated actual sites while in other cases we were given the name of 
district or others to contact for further information. Staff also phoned districts and other 
informants identified in this manner to solicit the names of schools with exemplary programs. The 
yield from roughly six months of intensive mail and phone contact was about 1 56 schools. Some 
of the schools were nominated for language arts, some for science, and some for mathematics. It 
was common for schools to be nominated for more than one curricular area. Table F-1 shows the 
number of schools nominated during site selection and the number identified by subject matter 
area. Schools could be nominated for more than one curricular area. 

This process was quite successful in identifying language arts sites but worked less well as a 
mechanism for identifying mathematics and science sites. The team relied on sources intimately 
familiar with a site for nominations for mathematics and science programs. Researchers often had 
to find a university center or other organization working to develop innovative mathematics and 
science curriculum in order to identify appropriate sites. The team also identified and contacted 
funding sources — such as the Eisenhower Mathematics and Science Education program — for 
names of sites with large populations of LEP students. 

The project’s research design called for the team to complete the nomination process in the 
fall of 1992 and conduct the initial fieldwork during the spring of 1993. In September 1992, 
during the process of site selection, the project staff received a stop-work notification from the 
Department of Education. Nine months later, in July 1993, the project was reinstituted by the 
Department of Education. The delay caused the project to lose some of the positive momentum 
that had been built and required that staff again create a sense of excitement in the field for the 
; 5 tudy and the benefits of participating The stop-work order also interrupted the site selection 
process and delayed it by more than the actual nine months of the work stoppage. In July, 
schools were closed and staff was unable to resume the site selection process until late in 
September 1993 At that point, staff had to make additional calls to many schools we had spoken 
to earlier and to call again many experts who had provided nominations. 
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Table F-1 

Nominated Sites By State and Subject Area 



State 


Language Arts 


Mathematics 


Science 


Total Schools 


Alaska 


2 


0 


0 


2 


Arizona 


4 


1 


4 


8 


California 


31 


17 


19 


58 


Colorado 


1 


1 


1 


1 


District of 
Columbia 


3 


0 


0 


3 


Florida 


3 


3 


5 


6 


Dlinois 


4 


0 


1 


4 


Maryland 


4 


2 


2 


8 


Massachusetts 


8 


5 


6 


11 


Michigan 


3 


0 


0 


3 


Nevada 


5 


0 


2 


5 


New Mexico 


2 


2 


2 


4 


New York 


9 


4 


4 


11 


North Carolina 


3 


1 


1 


4 


Ohio 


2 


1 


1 


2 


Texas 


7 


4 


9 


18 


Utah 


0 


4 


0 


4 


Virginia 


0 


2 


0 


2 


Wisconsin 


2 


0 


0 


2 


Total 


93 


47 


57 


156 



ERIC 



19 



( 



Selection Criteria. The next step in the process was to narrow the pool of 156 nominated sites 
to locate those that seemed to best fit the study criteria. Six selection criteria .vere established for 
exemplary sites; the selection criteria were developed with the assistance of the study advisors. 

Three selection criteria relate to the philosophy of school practices for language minority 
students. 

1. Innovation: The school alters traditional p. actices where necessary to sen/e the needs of 
language minority students. Tl’ere is an excitement about the school both inside and outside. 

2. Embedded: The practices for language minority students are not isolated, but are a part of 
the entire school program and are articulated with the type of practices used in earlier and later 
grades. In practice this means that the program is not limited to one teacher or one grade level 
The program spans more than one grade level. The program is part of larger commitment to 
educatiiig language minority students. 

3. High Expectations for LEP Students: the school's administrators and teachers have 
embraced and can articulate the philosophy of the program and share with parents and other 
community members a vision of a quality education for language minority students. The staff 
demonstrate an understanding of the societal context within which language minority students live 
and learn, and recognize the unique situations, challenge s and strengths of their students. 

Two criteria relate to implementation of the program. 

4. Longevity : the school's use of the identified practices is a serious long term effort. In 
practice, the school's program should have been in effect for two to three years. 

5. Qualified Staff : staffing and training of staff are appropriate to the practices being 
implemented. For example, if the program attempts to teach in Spanish, the teachers are bilingual 
in Spanish. 

The final criterion relates to the school's appropriateness for a national study. 

6. Generalizability: insights gained from studying the practices in place at selected school will 
be useful to other schools. In practice, this meant that the school serves students who are fairly 
typical of LEP students nationally, and has no special circumstance that makes it so unique that 
other schools cannot learn from their experience. 

Sites were also selected that fit geographic and programmatic diversity. 01 the 1 56 
nominated sites, approximately 60 percent were language arts sites and the remaining 40 percent 
were mathematics and/or science sites. The sites contacted were closer to 50 percent lang^aage 
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arts and 50 percent science and/or mathematics. Finally, some schools had multiple nominations 
and those sites were included in the next stage. 

Telephone Screening. Seventy-five candidate sites were chosen for additional contact based on 
the criteria just described. Telephone interviews were conducting using an instrument designed to 
capture basic demographic data on the school, a description of the progr im, staffing, evaluation, 
funding sources, and indicators of how the school would meet the selection criteria. Staff 
members called the district and school, interviewed either the principal or lead teacher for the 
program, and requested any additional written documentation on the program. When possible, 
additional informants from universities, training entities, or other sources were used to gather 
another perspective on the site. 

Staff held a one to one-and-one-half hour conversation with the principal or site coordinator 
at each of the 75 candidate schools. The results of the phone interviews were to reduce furt her 
the number of sites that had potential for further study. The telephone, screening process yielded 
25 potential fieldwork sites. 

Screening Visits. From the pool of 25 promising sites, demographic, geographic, and 
programmatic variables were used to select 1 5 sites for a one-day preliminary site visit to 
determine which programs would become the final case study sites. One-day visits by one to two 
fieldworkers to each of the 15 sites were designed to provide the research team with information 
that would allow tne selection of six to ten case study sites that best met study criteria. Prior to 
the visit, the study team had a great deal of information on the 15 fieldwork sites gathered from 
reports and articles, telephone interviews with informed experts, and site personnel. 

During the preliminary visits, fieldwork staff interviewed persons in responsible 
administrative positions at the district level and the site level as well as resource teacher(s), and 
classroom teachers. Classes were observed in the relevant curricular areas. The issues briefly 
explored in the preliminary visits include: 

1 . The design of the program: the purpose of the reform, the program's conceptual 
framework, curricuhjm, instructional strategies, materials, grouping strategies, and the 
role of research and research-based information in the design of the program. 

2. Implementation of the program, the forces and factors which influenced reform, 
program organization, staffing, and school climate. 

3 Impact of the program: evidence of improvements in student learning and previous 
program evaluations. 
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In this way we selected of the eight schools which we revisited for more intensive 6eld work It 
was impossible to find data on student outcomes that are comparable across the sites, particularly 
because LEP students are often not given the star. Jardized tests (in English) that districts or states 
require of most students.^ Therefore, we can not demons' ate quantitatively that the eight case stiidy 
sites are exemplary in the sense of evidence of significantly higher student achievement scores. 
Nevertheless, th e nomination, screening, and field visits all led to the conclusion that these schools are 
highly innovative and follow practices that are considered by researchers to provide outstanding 
learning opportunities for LEP and all students. 

Selected Sites. The eight sites chosen for intensive fieldwork represent diversity m geographic 
location, grade level stnicture, and native language of LEP students. Elementary schools included 
Del Norte Heights Elementary School in Ysleta Independent School District, Texas, Linda Vista 
Elementary School in San Diego Unified School District, California, and Hollibrook Elementary 
School in Spring Branch Independent School District, Texas. K through 8 schools included Inter- 
American School, Chicago, Illinois, and Graham and Parks Alternative School, Cambridge, 
Massachusetts. Middle Sc aools included Horace Mann Middle School, San Francisco, California, 
Hanshaw Middle School, Modesto, California and Wiggs Middle School in El Paso Independent 
School District, Texas. 

The schools were located in four states. Three schools were in California, one was in 
Massachusetts, one was in Illinois and three schools were in Texas. The languages represented in 
the LEP student populations included Spanish at three elementary schools, one K-8 school and all 
three middle schools. Haitian Creole speakers were served in one K-8 school. One middle school 
included both Spanish and Cantonese speakers. And one elementary school had students 
speaking a variety of South East Asian languages as well as Spanish The table below shows the 
demographic characteristics of the eight exemplary sites. 



‘ P. Bernun et al.. Meeting the Challenge of Language Diversity: An Evaluation of Programs for Pupils nith 
Limited Proficiency in English. ^ vols, (Berkeley, CA: BW Associates. February 1992), 
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Table F-2 

Dctnugrapbic Coiididuiis at the Elenieiitni^ Grade Case Study Sites 
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Table F-3 

Demographic Conditions at the Middle Grade Case Study Sites 
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Data Collection 



Intensive case study visits were conducted at eight sites; each of these sites were visited 
during the spring and summer of 1994. The intensive visits took place for three to four days with 
two to three team members per site. Research staff conducted interviews at the school, district 
and state level, as well as with external partners; led focus group discussions with students and 
parents at the study schools; and observed classrooms. 

Five team members compose the fieldwork team; Catherine Minicucci, Beryl Nelson and 
Katrina Woodworth. Paul Berman participated in two case study visits and Barry McLaughlin 
participated in one case study visit. At least one of the three senior researchers (Berman, 
Minicucci, and Nelson) visited each site and they were joined by at least one additional staff 
member. Spanish language classroom observation was performed by Katrina Woodworth. She 
also conducted focus groups in Spanish in order to gain better access to LEP students and their 
parents. A Creole-speaking observer, Faith Conant, was hired to observe the Haitian Creole 
classroom at Graham and Parks School. She was trained by study team members prior to the 
actual observation. 

Table E-4 displays the school and district personnel interviewed on site as well as the 
approximate length of each interview. 

Preparation for Case Study Visits. Prior to conducting case study visit<^ the research team 
piloted the instruments and protocols in a school environment similar to tnose to be studied. The 
team met to discuss the instruments and potential difficulties in administration. The coding of the 
classroom obser/ations for the case study visits required more extensive training. The 
observers/coders were members of the fieldwork team, who were trained on the classroom 
observation techniques. They observed videos of classroom interactions as they learned the 
coding scheme. Two observers coded the same class lessons in different subject areas and then 
compared the results to achieve uniformity in approach. In addition, fieldworkers reviewed 
written materials on the school's program provided by in advance, interview notes from previous 
contact with the site, and relevant literature. 

Classroom Observation. Classroom observations were conducted at the eight exemplary sites. The 
procedure for observing classes consisted of a pre-interview with the teacher, the observation itself 
which lasted fi'om 45 minutes to 1 hour and a half, and a post-observation inteniew with the teacher. 

The pre-observation intei-view with the teacher was used as an opportunity for the fieldworker to 
learn what would be taught during the class period. The number of students enrolled, the number of 
LEP students, the subject being taught, the goals of the lesson, how students would be organized for 
instruction, what activities had preceded this lesson were obtained. 



Table F-4 

Case Study Data Collection Sources 



Category 


Number 


Participants 


Length of 
Interview 


1. Interviews 


14 






a. District level 


5 


• Asst. Superintendent 

• Director of Curriculum 

• Director of Second Language 
Program 

• Evaluation Director 

• Business Officer 


• 45 minutes 

• 45 minutes 

• 1 hour 

• 30 minutes 

• 30 minutes 


b. External Agency 


1 


• Staff of agency assisting the 
school 


• 2 hours 


c. School 


8 


• Principal 

• 6 Teachers 

• Support Staff such as Counselor 


• 1.5 hours 

• 30 minutes 

• 30 minutes 


2. Focus Groups 


3 


• 1 Parent Focus Group 

• 2 Student Focus Groups 


• 30 minutes 

• 20 minutes 


3. Observations 


8 


• 4 Teachers 


• 1 hour 

• 2 consecutive 
days 
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The classroom observation examined how the instructional model is applied in practice in the 
classroom. The observation was coded into instructional segments. The segments followed the 
natural organization of the lesson and were divided up into how the students were grouped for 
instruction. For example, if a class lesson were going to consist of whole class discussion, followed by 
cooperative learning, followed by whole class discussion, this lesson would be coded as three 
segments. In most cases, two observers were present to record their observations of the class. 

Separate cooperative groups were coded as separate segments. 

The items coded included: 

1 . How the class was organized for instruction for each activity? 

2. What was the main task or activity being undertaken? 

3 . How is the primary language used and how is English used by students and teachers? 

4. How is technology used in the lesson? 

5 . How does the teacher manage the classroom? 

6. What materials are being used? 

7. What is the role of the teacher, whether directive, facilitative, or some other role? 

8. What is the role of the student, whether receiving knowledge, active in teacher led discussion or 
students working independently or presenting information? 

9 . What is the nature of the instructional script? 

10. What is the level of student participation? 

After the observation, a post-interview was held with the teacher to follow up on issues noted 
during the observation. Team members would inquire as to why a teacher structured a lesson in a 
certain way, or asked a student to perform a certain learning activity. These interviews were extremely 
valuable at providing deeper insight into classroom instructional practices. 

Observation notes were coded using a common format. They were incorporated into fieldnotes 
and summaries prepared by the study team. 

Interviews. Interviews were conducted with individuals at the district and level, as well as 

appropriate individuals in the external portion training institutions or interme e units which had 
a direct role in the design, implementation, or evaluation of the program. 

At the school site, one on one interviews were conducted with the principal, teachers (in addition 
to interviews relating to the observation of instruction), social workers, counselors, community/parent 
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liaisons, and nurses on school sites. Interviews were informal with questions asked of multiple 
informants. Results of interviews were compiled into field notes. 

At the district level, one on one interviews were conducted with the assistant superintendent, the 
second language program director, the curriculum director, the evaluation director, and a business 
officer. 

Personal interviews were conducted with the state director of second language programs 
when possible; when logistically impossible, a phone interview replaced the in-person interview. 

External partners working with the school were interviewed at their office locations away fi-om the 
school site. 

Focus Group Discussions. Two types of focus groups were conducted on site. The first was with 
students. Teachers were asked to assemble a group of 5 to 8 LEP students who had made significant 
progress in transitioning to English during the school year. Those students were taken to a quiet 
location away fi'om the classroom and asked some open ended questions about their experience at the 
school and what helped them learn science, mathematics, and English. The focus group was conducted 
in English or the primary language of students, depending on their fluency level. 

The second type of focus group was with parents of LEP students. School staff assembled 
parents in small groups for discussion. These groups were conducted mainly in the parents’ primary 
language. 

Results of the focus groups were compiled in fieldnotes and summaries. 

Documentation. Researchers gathered previously conducted program evaluations; data on 
student demographics, attendance, and achievement; and data on school resources. 

Data Analysis 

Fieldnotes and Summaries. Upon completion of fieldwork at a site, team members compiled their 
notes separately. One person was designated as lead person to assemble a comprehensive site 
summary including background on the school, the principal interview, the district interviews, the 
teacher interviews and classroom observations. Study team meetings reviewed the site summaries, 
involving all members of the fieldwork team to explore findings in the research. Discrepancies and 
conflicts in findings were discussed and resolved. If necessary, follow up phone calls were made to the 
sites to resolve remaining questions about the school's program. 

Case Studies. Fieldwork site summaries became the basis for the case study summaries prepared for 
Volume II: Case Studies. The Case Studies begin with a description of the school demographics and 
the community context. Next, they portray a scene fi’om one or more exemplary learning 
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environments. The remainder of each Case Study focuses on the exemplary aspects of the schools’ 
curriculum and instruction strategies, the program for LEP students, and the school structure. Finally, 
the Case Studies identify programs and policies at the district and state level that support the 
implementation of exemplary practices. Each Case Study includes two figures. The first one illustrates 
key instruction and curriculum features and the design of the language development program for LEP 
students. The second figure depicts significant features of the school structure and ways in which the 
district supports the school. Each school’s Case Study was submitted to the school’s principal (and in 
some cases a lead teacher) for review and permission to use the school's. In all cases, schools agreed 
to have their name published in the reports to OERI. 

Cross-Site Analysis. Once the case study volume was completed and submitted to OERI and 
participating sites for review, the team set about preparing the cross-site analysis. The research team 
prepared matrices for each of the five Case Study Research Areas; 1) The Context for Reform; 2) The 
Design and Implemntation of the Reform (including restructuring, curriculum and instruction, and 
language development programs); 3) The Role of Research-based Information in the Reform, 4) 
Resources Required for the Reform, and 5) Impact of the Reform. The eight case study sites were 
described in terms of each of the Operational Elements for Data Collection (see Tables E-5 through E- 
9) for each of the five Case Study Research Areas. These tables are included as an Appendbc to 
Volume I: Findings and Conclusions. 

The next step of the cross-site analysis was to develop tables with the key.features at each school 
that surfaced in the process of writing the Case Studies. As mentioned above, each of these features 
was highlighted on two figures presented in the Case Studies. The tables focused on the key features 
in two areas; 1) curriculum and instruction, and 2) school structure. This process allowed the 
researchers to identify strategies that were common across sites or unique to a particular site. Based 
on these tables, strategies that were used at multiple sites (i.e., integrated curriculum and schools- 
within-schools) were identified and more tables were created to illustrate the variety of ways that 
schools at a pven site implemented a particular strategy. (For example, schools integrated curriculum 
using a variety of stratepes including project-based learning and thematic units taught by a team of 
teachers.) 

External support fi"om an external partner, the district, and the state was analyzed through a 
similar process of identifying approaches, programs, and policies that were common across the 
exemplary sites or unique to a particular site. The study team focused on policies that directly and 
positively impacted the case study schools. 

The results of the cross-site analysis are presented as the study findings in Volume I: Findings and 
Conclusions. 



